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TO ALL WHOM IT MAY CONCERN: 

Be it known that David T. Chen of 12 Sunset Road, 
Somerville, MA 02144, Steven D. Pieper of Gove Hill Road, 
Thetford Center, VT 05075 and Michael A. McKenna of 232 
Broadway, Cambridge, MA 02139, have invented certain 
improvements in VIDEO-BASED SURGICAL TARGETING SYSTEM of 
which the following description is a specification. 
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Reference To Pending Prior Applications 
This application: 

(1) is a continuation in part-of -pending prior 
U.S. Patent Application Serial No. 09/746,992, filed 
12/21/2000 by David T. Chen et al . for VIDEO-BASED 
SURGICAL TARGETING SYSTEM (Attorney's Docket No. 
MMS-4 CON 2), which patent application is hereby 
incorporated herein by reference; and 

(2) claims benefit of prior pending U.S. 
Provisional Patent Application Serial No. 60/277,664, 
filed 03/21/01 by David T. Chen et al. for VIDEO-BASED 
SURGICAL TARGETING SYSTEM (Attorney's Docket No. MMS-B 
PROV) , which patent application is hereby incorporated 
herein by reference. 
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Ficld Of The Invention 

Tlus invention rclalcs to surgical syslcnis in general, 
lo surgical targeting systems. 



Rackground Of The Invention 

Many medical procedures must be cairied out at an interior site which is 
normally hidden from the view of tlie physician. In these situations, the 
physician tyi)ically uses some sort of scanning device to 



anatomy at the interior site prior to, and in preparation for, conducting the 
actual procediue itself. These scannijig devices typically ijiclude MRI devices, 
CT scanners, X-ray machines, ultrasound devices and the like, and serve to 
provide the physician with a preliminary knowledge of the patient's intenial 
anatomical stiucture prior to commencing the procedure. The physician can 
then use this infonnation to plan die procedure in advance, taking into account 
patient-specific anatojnical structure. In addition, the physician can also use 
the information obtained from these scans to more precisely identify the 
location of scleclcd slmctures (e.g. tumors and llie like) which may themselves 



and more pailicularly 



examine the patient's 
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be located wilhia inlenial organs or oilier internal body stiuctures. As a 
result, the physician can more easily "zero in" oi\ such selected structures 
during the subsequent procedure^ with less trauma to host organs or other 
internal body stiiictui es. Fuithennore, in many cases the sliiictures of interest 
may be quite small and dilTicult to identify with llic naked eye. In these cases, 
preliminaiy scanning of the patient's internal anatomy using high resolution 
scanning devices can help the physician locate such slmctures during the 
subsequent procedure. 

In general, scanning devices ofthe sort described above tend to generate 
a series of two-dimensional (i.e., "2-D") images of the patient's anatomical 
structure. For example, CT scanners generate 2-D images which are viewed 
directly by the physician. By viewing a plurality of these 2-D images, the 
physician can mentally generate a three-dimensional (i.e., "3-D") sense of the 
patient's anatomical stmcture. 

Some scanning devices create large numbers of 2-D images during the 
scanning process, with each 2-D image representing a plane or slice taken 
tlirough tlie scanned stiucture. Fuitliermore, some scanning devices also have 
associated computer hardware and software for building a 3-D computer 
model of tlie patient's scanned sti'ucture using a plurality of these 2-D images. 
For example, some Mill devices and some CT scanners have such associated 
computer hardware and software. In these cases, an operator using this 
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scanning equipment and associated computer hardware and soAware can 
create a number of 2-D images, assemble Uiem into a 3-D computer model of 
the scanned stiojcture, and then generate various images of that 3-D computer 
model as seen from various angles so as to enliance understanding of the 
patient's anatomical stiucture. 

WIulc the infonnation generated by the nforcmcnlioncd scanning devices 
is generally of great benefit to physicians, certain significant limitations still 
exist. For one thing, it can be veiy difficult for a physician to physically 
locale a particular analonn'cal sti-ucturc during a medical procedure, even when 
tliat slructuic is readily identifiable in a scanned image.: Hiis may be because 
tlie structure is vciy small and difficult to sec with the naked eye, or because 
it is not readily visible against a particular background, or because it is itself 
located within an internal organ or other internal body stiucture, etc. For 
anollier tiling, even when the stnictui e of interest is successfully located by the 
physician, it can sometimes still be extremely difficult for the physician to 
reliably direct a medical instilment to that stiucture. This may be because the 
structure is quite small and difficult to target accurately, or because 
intervening body stiucture mandates a complex approach, etc. 
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Objects Of The Invention 

Accordingly, one object of Ihe present invention is to provide a surgical 
targetmg syslcni to facilitate locating a particular anatomical structure during 
a medical procedure. 

Anotlicr object of the present invention is to provide a video-based surgical 
targeting system to facilitate locating a particular anatomical structure during 

a medical procedure. 

And another object of the present invention is to provide a video-based 
surgical targeting system which pennits a series of patient-specific 2-D 
images (obtained by scanning patient anatomy using one or more scanning 
devices of the type described above) to be assembled into a 3-D computer 
model of the patient's scanned structure. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which allows a physician to view the aforementioned 
patient-specific 2-D images on a display in any desired access sequence. 

And another object of the present invention is to provide a video-based 
surgical targeting system which allows a physician to assemble a series of 
patient-specific 2-D images into a patient-specific database, and then to 
generate viilual images from the aforementioned patient-specific database, as 
seen from any desired virtual camera position, for viewing on a display. 

Yet another object of the present invention is to provide a video-based 
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surgical targeting system which allows a physician to generate virtual images 
from the aforementioned 3-D computer model, as seen from any desired 
virtual camera position, for viewing on a display. 

And another object of the present invention is to provide a video-based 
sujgical targeting system which permits a physician to place virtual planning 
markers about any sites of interest while viewing one or more of the 
aforementioned patient-specific 2-D images, with those virtual planning 
markers tl)cn being incoii)oraIcd into the 3-D computer model, whereby those 
virtual planning markers can be displayed in their appropriate 3-D positions 
when generating virtual images of the 3-D computer model. * 

Still another object of the present invention is to provide a video-based 
surgical targeting system which pennits a physician to place virtual planning 
markers about any sites of interest while viewing virtual images of the 3-D 
computer model, with lliose virtual plamiing markers then being incoiporated 
into tlie 3-D computer model, whereby those virtual planning markers can be 
displayed in their appropriate positions wlien generating subsequent virtual 
images of the 3-D computer model. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which pennits a physician to place virtual planning 
markers about any sites of interest while viewing virtual images of the 3-D 
computer model, with those virtual planning markers then being incoiporated 
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into tlie 3-D computer model and into the database of 2-D images, whereby 
those viitxial planning markers can be displayed in their appropriate positions 
when subsequently generating virtual images of the 3-D computer model or 
when subsequently displaying 2-D images from the patient-specific database. 

Yet anotlicr object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be displayed lo a physician, and which 
permits a virtual image generated from the 3-D computer model lo be 
displayed to a physician, according .to the directive of the physician. 

Anollier object of llie present invention is to provide a video-based surgical 
targeting system which pcnnits a real image obtained by a real-time imaging 
device (e.g. a video camera) to be appropriately merged with a corresponding 
virtual image generated from the 3-D computer model. 

And another object of the present invention is to provide a video-based 
surgical targeting system which pennits a real image obtained by a real-time 
imaging device (e,g. a video camera) to be appropriately merged with a 
coiresponding virtual iinage generated from llie 3-D computer model, whereby 
the two images will be in registration with one another. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
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virtual image generated from Ihe 3-D computer model, wJiereby tlie Iwo 
images will be in regislraliou willi one anollier, and whereby the physician can 
choose lo display either one of the two images exclusive of the other, or a 
composite of both images simultaneously. 

Yet another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a conesponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby the physician can modify the virtual image generated from the 3-D 
computer model as needed, by clipping or fading, so as to expose the viilual 
planning markers to view, with the virtual planning markers being 
superimposed on the real image generated by the real-time viewing device. 

And another object of the present invention is to provide a video-based 
surgical largetuig system which pennits a real image generated by a real-lime 
imaging device (e.g. a video camera) to be mcigcd with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registialion with one another, and 
whereby the physician can modify the viilual image generated from the 3-D 
computer model so as to expose only the virtual planning markers to view, 
with the virtual planning markers being superimposed on the real image 
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generated by the real-time imaging device. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which pcnnils a real image oblaincd by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simuhaneously displayed in registration with one another, and 
whereby this registration will be automatically maintained even as the 
real-time imaging device is moved about relative to the anatomical site, with 
tlie virtual image being automatically generated so as to follow the real image. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D coniputer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby tlus registration will be automatically maintained through the use of 
a computerized position and orientation tracker connected to the imaging 
device even as the real-lime imaging device is moved about relative to the 
anatomical site, with the virtual image being automatically generated so as to 
follow the real image. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which pcnnils a real image obtained by a real-time 
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imaging device (e.g. a video cajnera) to be merged wiili a conesponding 
virtual image generated from ihe 3-D computer model, whereby Ihc two 
images will be simultaneously displayed in registration with one another, and 
whereby tliis regisliation will be automatically maintained through the use of 
a computer search algoritlnn based on the real image and the virtual image 
even as tlie reaMime imaging device is moved about relative to the anatomical 
site, with the virtual image being automatically generated so as to follow the 
real image. 

Yel another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby a surgical instrument can be tracked along with the real-time imaging 
device in order tliat tlie suigical insliiiment can be guided about the anatomical 
site using the virtual image generated from the 3-D computer model. 

And another object of the present invention is to provide a video-based 
surgical targeting system which permits a virtual image to be generated from 
the 3-D computer model, and further wherein this virtual image can 
correspond to an "over the shoulder" view of the working tip of a surgical 
instrument located at the anatomical site. 
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And another object of the present invention is to provide an improved 
method for locating anatomical stiuctures during a medical procedure. 

Summary Of The Invention 

Tliese and other objects of the present invention are achieved tluougli the 
provision and use of a novel video-based surgical targeting system. 

In one fonn of the invention, the video-based surgical targeting system 
comprises a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patieni; a paticnt^spccific 3-D computer model 
of the anatomical sti-uclurc of the patieni, the patient-specific 3-D computer 
model being assembled from the plurality of 2-D images contained in tlie 
patient-specific database; marker placement means for (i) inseiling viilual 
planning markers into the 2-D images contained in the patient-specific 
database, and/or (ii) adjusting the positions of virtual planning markers 
inserted into the 2-D images contained in the patient-specific database and 
thereafter incoiporaled into the patient-specific 3-D computer model, or 
inserting virtual planning markers into the 3-D computer model; an image 
generator for generating a virtual image of the anatomical structure modeled 
by the patient-specific 3-D computer model; real-time image generating means 
for generating a real image of the anatomical stiiicture of a patient; video 
mixing means for mixing the viilual image and the real image into an output 
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image, whereby the output image may comprise eillicr one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying the output image. 

Preferably the video-based surgical targeting system also includes 
registration means for placing the virtual image in registration with the real 
image. These registration means may include means for manually aligning the 
virtual image with the real image, or means for automatically aligning the 
virtual image with the real image. The registration means may also include 
tracking means for tracking the position of the real-lime image generating 
means. 

Preferably, the video-based surgical targeting system further comprises 
instrument tracking means for tracking the position of a surgical instrument. 

In an alternative fonn of the invention, the video-based surgical targeting 
system comprises a patient-specific database comprising a plurality of 2-D 
images of the anatomical stiuclure of a patient; marker placement means for 
insetting viilual planning nicukcrs into tlie 2-D images contained in the patient- 
specific database; an image generator for generating a viilual image of the 
anatomical stioicture represented by the plurality of 2-D images contained in 
tlie patient-specific database; real-time image generating means for generating 
a real image of the anatomical structure of a patient; video mixing means for 
mixing the viilual image and the real image into an output image, whereby the 
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output image may comprise either one of ihe two images exclusive of the 
other, or a composite of both images; and display means for displaying the 
output image. 

In another alternative fonn of the invention, the video-based surgical 
targeting system comprises a palient-spccific database comprising a plurality 
of images of the anatomical stiuclure of a patient; a palicnt-spccific 3-D 
computer model of the anatomical structure of the patient, the patient-specific 
3-D computer model being assembled from the plurahly of iniagcs contained 
in the patient-specific database; marker placement means for (i) inserting 
viitual planning markers into the images contained in tlie patient-specific 
database, and/or (ii) adjusting the positions of virtual planning markers 
inserted into the images contained in the patient-specific database and 
thereafter incorporated into the patient-specific 3-D computer model, or 
inserting virtual planning markers into the 3-D computer model; an image 
generator for generating a viilual image of the anatomical structure modeled 
by tlie patient-specific 3-D computer model; real-time image generating means 
for generating a real image of the anatomical structure of a patient; video 
mixing means for mixing the viilual image and the real image into an output 
image, whereby the output image may comprise either one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying said output image. 
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In yet anolher alternative fonn of ll)e iiivciilion, the video-based surgical 
targeting system comprises a patient-specific database comprising a plurality 
of 2-D images of the anatomical structure of a patient; marker placement 
means for inserting virtual planning markers into the 2-D images contained in 
tJic patient-specific database; an linage generator for generating a virtual image 
of llie anatomical sU-ucture defined by the paticnl-specific database; real-time 
image generating means for generating a real image of llie anatomical structure 
of a patient, the real-time image generating means being adapted so as to 
dynamically update the patient-siiecific database via the real image; video 
mixing means for mixing the virtual image aiid the real image into an outj)ut 
image, \yhcreby the output image may comprise cither one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying said output image. 

Hie present invention also comprises a method for targeting an anatomical 
structure using the video-based surgical targeting system. 

Brief Desc ription Of The Drawines 

These and other objects and features of the present invention will be more 
fully disclosed or rendered obvious by tlie following detailed description of the 
prefeired embodiments of the invention, which are to be considered together 
witli Uie accompanying drawings whercni like numbers refer to like parts and 
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furllier wherein: 

Fig. 1 is a schematic view of the major components of a video-based 
surgical targeting system formed in accordance with the present invention; 

Fig. 2 is a view of an excniplaiy palienl-spccific 2-D image of the sort 
contained in the patient-specific database; 

Fig. 3 is a view of an exemplaiy patient-specific 2-0 image of the sort 
contained in the patient-specific database, wlicrcin the image has had several 
virtual planning markers placed into the image; 

Fig. 4 is a composite view of (i) a real im<nge obtained by a real-time 
imaging device (e.g. a video camera), and (ii) a virtual image generated from 
the 3-D computer model, wherein the two images arc not yet in registration 
with one another; 

Fig. 5 is a composite view of (i) a real image obtained by a real-time 
imaging device (e.g. a video camera), and (ii) a virtual image generated from 
the 3-D computer model, wherein the two images have been placed in 
registration with one another; 

Fig. 6 is a view like that of Fig. 5, except that the virtual image generated 
from the 3-D computer model has been sliced away to expose the virtual 
planning markers; 

Fig. 7 is a view like that of Fig. 5, except that the virtual image generated 
from the 3-D computer model has been faded away to expose the virtual 
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planning markers; 

Fig. 8 is a schematic view showing Ihe positional rclationsliips between 
various elements oftlie video-based surgical targeting system and the patient's 
anatomical sliucture, as well as the positional relationship between an 
exemplaiy surgical instrument and the patient's anatomical stiucture; 

Fig. 9 is a viilual image generated from the 3-D conipiilcr model, showing 
a viilual "over the shoulder" view from the tij) of a tracked surgical insliiiment; 
and 

Fig. 10 is a flowchart illustrating one way of operating a video-based 
surgical targeting system fonncd in accordance with the present invention. 

Detailed D escription Of The Prcfcn cd Embodiments 

Looking first at Fig. I, the stalling point for the present invention involves 
a data acquisition device 5 which is adapted to generate patient-specific 2-D 
images. For example, data acquisition device 5 might comprise an MRI 
device, a CT scanner or any other scanning device of the sort adapted to 
provide a series of 2-D images of the patient's anatomical structure, where 
each 2-D image corresponds to a plane or slice taken through the scanned 
structure. Alternatively, data acquisition device 5 miglU comprise an X-ray 
macliine or other imaging machine adapted to provide a set of 2-D images of 
tlie patient's anatomical stmcture, where each 2-D image corresponds to a data 
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set relating (o the scanned slmcluie. 

Tlie palicnt-speciflc 2-D images gencialcd by dala acquisition device 5 are 
stored in a patient-specific database 10. Patient-specific database 10 is 
constructed so that the patient-specific 2-D images can be accessed 
individually or in any pailicular sequence desired. Preferably patient-specific 
database 10 comprises an appropriate set of dala contained in a computer 
storage system. 

Tlie patienl-'spccific 2-D images stored in patient-specific database 10 are 
used in building a patient-specific 3-D computer model 15. Patient-specific 
3-D computer model 15 may be built from the infonnation stored in 
paticnl-spccific database 10 using any one of the many algorithms well knov^n 
in the art, so long as the patient-specific 3-D computer model is capable of 
generating virtual images of the patient's anatomical structure from 
substantially any location where a real camera niay be positioned relative to 
that anatomical stiucture during a given procedure. Alteiuativcly, patient- 
specific 3-D computer model 15 may be built from the infonnation stored in 
patient-specific database 10 using any one of tlie many algorithms well known 
in the art, where the patient-specific 3-D computer model is capable of 
generating virtual images of the patient's anatomical structure from a 
predetermined set of possible virtual camera positions. Preferably, 
patient-specific 3-D computer model 15 is conslrtjctcd so that it is capable of 
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generaling virtual images of the patient's anatomical stiucture from any 
location whatsoever, with any desired field of view, and including any otlier 
camera-specific criteria desired (e.g. particular camera characteristics such as 
focus, optical characteristics, optical distortions, etc.). By way of example, 
one possible algorithm for building the patient-specific 3-D computer model 
out of the information stored in the patient-specific database 10 might be the 
Marching Cubes algorithm. 

In one prefeiTcd embodiment of the invention, patient-specific 3-D 
computer model 15 comprises ai) appropriate computer software model 
resident on an appropriate digital computer. Patient-specific 3-D computer 
model 15 is preferably constructed using a plurality of polygonal models (o 
model tlie patient's real anatomical structure. As is well known in the ail, such 
polygonal models generally comprise a collection of points defining the 
surface of tlie 3-D computer model, along with some connectivity information 
relating to how these surface points are connected to one another. Preferably 
there is one polygonal model per anatomical stmcture being modeled. Of 
course, the choice of which particular computer software model is used to 
model a particular anatomical structure ultimately depends on a variety of 
factors, such as the particular anatomical structure being modeled, the 
computer hai dwaie available, the volume of data to be handled, the particular 
medical procedure which is to be conducted, etc. 
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Of course, it is also anlicipaled lhat die palienl-spccific 3-D computer 
model could be dynamic as well as static. For example, dynamic changes 
could occur in the model as a result of computer simulation (e.g. through the 
use of a Finite Element Model) or because more current real-time data (e.g. 
from video camera 45) is used to update the model, or bolli. 

It is also anticipated lhat patient-specific 3-D computer model 15 might 
include data from sources other than llie 2-D images contained in 
patient-specific database 10. For example, patient-specific 3-D computer 
model 15 might include infonnalion bbtained from a 3-D surface digitizer such 
as that used in certain types of cranial surgeiy, or patient-specific 3-D 
computer model 15 might include infonnalion obtained from a temperature 
probe or a chemical probe, hi fact, in addition to the 2-D image data obtained 
from patient-specific database 10, patient-specific 3-D computer model 15 
might include substantially any type of infonnation gathered by almost any 
type of device or sensor or process. 

An integral pait of the present invention involves tlie placement of virtual 
planning markers into tlie patient-specific data, using apparatus 20. This may 
be done by placing such virtual planning markers into one or more of the 
patient-specific 2-D images contained in patient-specific database 10 before 
patient-specific 3-D computer model 15 is created; or by placing such virtual 
planning markers into patient-specific 3-D computer model 15 after that model 
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has been created; or by some combination of the two. 

More paiticularly, apparatus 20 preferably comprises computer hardware 
and soflware adapted to allow a physician to access one or more of the 
patient-specific 2-0 images c(|ntaincd in patient-specific database 10 and 
present Uicm for viewing, in the manner showi in Fig. 2. Then, using a mouse 
or other data cntiy device, the j)hysician can place one or more virtual 
planning markers 25 into an ;icccsscd palicnt-spccific 2-D image, in the 
manner shown in Fig. 3. Those virtual planning markers 25 can consist of 
substantially any geometric forn such as a point, a line, a circle, a plane, a 
path (either straight or curved), etc., and are positioned about anatomical 
structures of pailicular interest to lIjc physician. For example, virtual planning 
markers 25 inight be placed abou a suspected tumor, or they might be placed 
about particularly sensitive anatomical sti-ucturcs such as vascular tissue or 



nerve bundles, etc. Naturally, 
paiticular virtual planning marker 
the particular anatomical sliucture 



associated 2-D image. As a result, 



he particular geometric fonn chosen for a 
may depend on a variety of factors, such as 
: involved, the compuler hardware available, 
the volujne of data to be handl ed, the particular medical procedure to be 
conducted, etc. Regardless of the particular fonn of virtual planning marker 
chosen, each virtual planning mjirker 25 which is placed on the 2-0 image 
using apparatus 20 is then stored in patient-specific database 10 along with its 

when palicnt-spccific 3-D computer model 
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15 is subsequently created fiom llie data contained in patient-specific database 
10, patient-specific 3-D computer model 15 will incojporate virtual planning 
markers 25 into the model. 

Alternatively, a physician may use apparatus 20 to access patient-specific 
3-D computer model 15 and then, using a mouse or other data cntiy device, 
adjust the position of one or more of the virtual planning markers 25 which 
were previously placed into patient-specific database 10 (and thus 
incorporated into patient-specific 3-D computer model 15 when that 3-D 
computer model was created). Tlie adjusted positions of these virtual planning 
markers 25 are then stored in patient-specific 3-D computer model 15. 
Furtlicnnore, an updated 2-D image of these virtual planning markers can then 
be incorporated into the patient-specific database 10. 

Fuilhennore, where patient-specific 3-D computer model 15 has not 
already had a particular virtual planning marker 25 incorporated therein, a 
physician may also use apparatus 20 to access patient-specific 3-D computer 
model 15, and then use a mouse or other data entiy device to insert one or 
more viilual planning markers 25 directly into patient-specific 3-D computer 
model 15. 

Once virtual planning markers 25 have been properly positioned in 
patient-specific 3-D computer model 15, virtual images incorporating these 
virtual planning markers can be created by an image generator 30, fed through 
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a video mixing device 35, and then presented to the physician on display 40. 

At llic same lime, real images obtained from a video camera 45 can be fed 
llirough video mixing device 35 and tlicn presented to the physician on display 
40. 

More pailiculaily, image generator 30 comprises computer hardware and 
software of the sort well known in the art for generating virtual images from 
patient-specific 3-D computer model 15. 

Video mixing dcxicc 35 comprises a video mixing device of the sort well 
known in the art, whereby the surgeon can selcclivcly display (i) a virtual 
image created by image generator 30 from patient-specific 3-D computer 
model 15, (ii) a real image obtained from video camera 45, or (iii) a composite 
of the virtual image and the real image, where the virtual image is 
superimposed against the real image. 

In order for the foregoing composite image to be more useful lo the 
physician, the virtual image generated from patient-specific 3-D computer 
model 15 should be placed into regisUalion with the real image obtained from 
video camera 45. This is done in several steps. 

First, and looking now at Fig. 4, a virtual image 50 is created by image 
generator 30, fed through video mixing device 35, and presented on display 
40. Simultaneously, a real image 55 is captured by video camera 45, fed 
llirough video mixing device 35, and presented on display 40. Video mixing 
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device 35 is ananged so as to present virtual image 50 and real image 55 
simultaneously on display 40, with the virtual image being superimposed on 
tlie real image (i.e., so that the virtual imnge is in the foreground and the real 
image is in the background, in the manner shown in Fig. 4). 

Second, and looking now at Fig. 5, the two images are placed into proper 
registration with one another. Wis image registration can be accomplished in 
either one of two ways. 

A first, and generally, more preferable, tcchniciue involves holding the 
position of video camera 45 (and hfcncc real image 55) constant and moving 
the position of the "virtual object" or the "virtual- camera" by means of 
apparatus 57 until the virtual image 50 is brought into registration with real 
image 55. This can be done automatically by having image generator 30 use 
a search algonlhm to match the virtual image to the real image, in which case 
apparatus 57 includes computer hardware and so/lwarc of the sort well known 
in tlie art to cause image generator 30 to work through a search algoriilun to 
match tlie virtual image to the real image; or it can be done manually, in which 
case apparatus 57 includes computer hardware and soAware of the sort well 
known in the art to allow the physician to drag the virtual image 50 into 
registration with real image 55, using a mouse or other data entry device 
attached to image generator 30. One convenient way that mouse motion can 
be used to conlrol ilic 3-D niovcnicnls of Ihc palicnl-spccific model is to map 
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such motion into vectors defined by llie view direction of the virtual camera, 
as will be well known to persons skilled in the ail. 

A second, and generally less preferable, technique involves holding the 
virtual image 50 constant and moving video camera 45 (or the patient) until 
real image 55 matches virtual image 50. 

Regardless of which technique is used, once virtual image 50 h<is been 
matched to real image 55, the position of the "virtual camera" (i.e., the location 
from which the virtual image appears to be seen) will be matched to the actual 
position of video camera 45. 

Once virtual image 50 has been placed into proper registration with real 
image 55, image generator 30, video camera 45 and video mixing device 35 
can be used to present the virtual and real images on display 40 in various 
presentation fonnats so as to facilitate a particular medical procedure. In 
particular, one can use image generator 30, video camera 45 and video mixing 
device 35 to superimpose a virtual image (generated from patient-specific 3-D 
computer model 15) against a real image (generated by video camera 45), with 
image generator 30 being directed to modify the virtual image so as to expose 
one or more of the virtual planning markers 25 present in patient-specific 3-D 
computer model 15, whereby the anatomy highlighted by virtual plamiing 
markers 25 will be brought quickly to the attention of the physician. 

In this respect it is to be appreciated that since virtual planning markers 
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25 will fiequenlly be located witliiii iiilerior poilions of Ihe sliuclure modeled 
by patieiU-specific 3-D coinpuler model 15, exposing llie otherwise-hidden 
virtual planning markers to view will involve rendering some or all of the 
virtual anatomical sU-uctuie tiansparent or semi-transparent. Tiiis can be done 
cither by (i) slicing away any portions of tlje virtual image required in order 
to expose virtual planning markers 25, whereby those virtual planning markers 
will be rendered visible against the background real image, in the manner 
shown in Fig. 6; or (ii) fading away some or all of the virtual image so as to 
expose such virtual planning markers against the real image, in the manner 
shown in Fig; 7. As a result, a physician can then conduct a medical 
procedure willi die confidence of using real images generated by video camera 
45, while having virtual planning markers 25 superimposed against the real 
image of the anatomical stmcture so as to help guide the procedure. 

While the foregoing system constitutes a major improvement over the 
prior art, it has also been recognized that it can itself be improved upon 
significantly by adding camera tracking means to video camera 45. 

More particularly, in many medical procedures, the position of video 
camera 45 may change on a fairly frequent basis. Since it is important for 
virtual image 50 (generated from patient-specific 3-D computer model 15) to 
remain in proper registration with real image 55 (obtained from video camera 
45), it is ncccssaiy for the video-based surgical targeting system to reestablish 
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proper correspondence between Uie Iwo images each time the video camera 
moves. As noted above, this proper concspondcncc can be rccslabh'shcd each 
time video camera 45 moves, by citlicr (i) Iiaving the physician manually drag 
virtual image 50 into registration with real image 55, using a mouse or other 
data enuy device attached to image generator 30, or (ii) liaving the system use 
a search algorithm to match the virtual image to the real image. While both of 
tliese procedures are generally capable (assuming that the anatomical structure 
remains slationaiy) of accurately yielding the desired image correspondence, 
each one suffers from a significant disadvantage. In the first case, requiring 
tlie physician to physically drag Uie viilual image into registration with the real 
unage each lime Uie video camera moves can be inconvenient where the video 
camera is moved about frequently. In the second case, it can take the system 
a substantial amount of time to solve for the coirecl image correspondence 
when using a search algorithm, even where high speed hardware is utilized. 
Even where speed is not a problem, automated search strategies of the type 
generally used are prone to eiTors through false matches. 

Accordingly, it has been recognized that by putting a tracker on video 
camera 45, the position of the real camera can be continuously monitored, 
whereby each time the real camera moves, the position of the virtual camera 
can be conespondingly adjusted by image generator 30. In this way, the 
virtual image generated by image generator 30 cnn quickly and easily be 
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maintained in regislralion willi ihe real image, regardless of how often video 
camera 45 is moved. 

More particularly, and looking now at Fig. 8, there is shown a video 
camera 45, an anatomical structure 60, and a tracker system 65. Tracker 
system 65 comprises a tracker 70 winch is attached to video camera 45, and 
a tracker base 75 wliich defines the coordinate system of the H acker system. 

In this setting, M,^ can be considered to represent the matrix 
transfonnation from the patient's anatomical stmcture 60 to camera 45; Ma 
can be considered to represent the matrix transfonnation from camera 45 to 
tracker base 75; and M^^ can be considered to represent the matrix 
transfonnation from anatomical structure 60 to tracker base 75. 

McT is known from the tracker system. Furthennorc, once the virtual 
image generated by image generator 30 has been placed in registration with the 
real image generated by camera 45, the virtual camera position will be known 
relative to tlie virtual anatomical stiojcture, and hence the real camera position 
will be known relative to tlie real anatomical stmcture. Thus, real matrix Mpc 
will also be known, hi addition, since and Mpc are then both known, it is 
possible to solve for M,,^. Accordingly, the position of anatomical structure 
60 will then also be known within the relative coordinate system defined by 
tlie tracker system. 

In view of the foregoing, the virtual image generated by image generator 
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30 can be quickly and easily maintained in registration with the real image, 
regardless of how often camera 45 is moved. 

Fuillicnnorc, another tracker 80 can be positioned on a suigical instrument 
85. In tliis case, tracker 80 will provide M-n, where Mti represents the matrix 
transfonnation from Uackcr base 75 to instrujncnt 85. Accordingly, once M„ 
and Mti a' c known, it is possible to solve for M,.„ where M,., represents the 
matrix Uansfonnation from the patient's anatomical structure 60 to insli-ument 
85. With M,., known, it is then possible to track the position of surgical 
inslj-umcnl 85 relative to anatomical structure 60. This is a vciy powerful tool, 
since it allows a virtual representation of surgical instrument 85 to be 
accurately added to the properly registered virtual and real images presented 
on display 40, i.e., it allows a virtual representation of instrtiment 85 to be 
shown in proper registration with a real image of anatomical structure 60 on 
display 40. 

Furtliermore, as the position of surgical instrument 85 is tracked relative 
to anatomical structure 60, it is also possible to provide a viitual image of 
surgical instrument 85 as that surgical insli-umenl 85 moves through the 
anatomical stiucture, even when some or all of that instrument might be 
hidden from the view of video camera 45. For example, and looking now at 
Fig. 9, it is possible to generate a viilual "over the shoulder" view of the distal 
tip of surgical insli-umcnt 85 moving through anatomical stiucturc 60, where 
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that tip will be shown in proper correspondence (o various structures, e.g. a 
tumor 95, vascular structures 100, etc. It is also possible to generate an 
additional virtual object 90 showing a line extended along the axis of the 
surgical instrument 85 so that the physician can see where the surgical 
instiiunent would go if moved furtlicr along its cun cnt trajectory. Of course, 
since the viilunl image generated by the video-based surgical targeting system 
will then differ significantly from the real imngc generated by video camera 
45, video mixing device 35 should be directed lo totally suppress the real 
image generated by video camera 45 so that it will no longer be shown on 
display 40, in order to avoid confusing the physician. 

Modifications Of The Preferred Embodiments 

It is, of course, possible to modify the preferred embodiments disclosed 
above without dcpailing from the scope of the present invention. 

Thus, for example, while only one camera 45 has been disclosed above, 
it should be appreciated that two or more cameras 45 may be used. 
Furthermore, it should also be appreciated that camera 45 may comprise a 
video camera, or it may comprise an endoscope, or it may comprise some 
other type of real-time image capturing means, e.g. it may comprise an 
ultrasound device. 

Additionally, it should be recognized that the system might be adapted so 
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diat patient-specific database 10 comprises non-2-D images, e.g. database 10 
might comprise one or more images generated by a Cybcrware 3-D scanner. 

Allemativcly, patient^specific 3-D computer model 15 might be omitted 
from tJic appaialus entirely. In Uiis case, the virtual images could be generated 
from patient-specific database 10 by image generator 30 through the use of a 
volume rendering procedure of the soil well known in the art. 

It is also anticipated that one could use a tracked surgical instrument 85 
to determine the location of anatomicil structure 60. This can be 
accomplished by using the tracked surgical instrument 85 to engage knov^n 
fiducial points on the anatomical structure. Alternatively, a tracked surgical 
instrument 85 can be used to sample multiple surface points located on the 
anatomical structure and then use a data matching procedure to correlate the 
sampled points vk'iih cither patient-specific database 10 or patient-specific 3-D 
computer model 15. By way of example, a least squares fit might be used to 
correlate the sampled points with the 3-D computer model. 

Also, it should be appreciated llial Uic video-based targeting system could 
be used to target objects in non-medical applications, e.g. it could be used to 
target objects concealed within the interior of complex machines, or objects 
(e.g. a conduit) concealed beneath surface of a structure (e.g. the floor of a 
building). 

Furllicnnorc, it is possible to place tracking means diicclly on the patient 
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SO as to track the position of the patient's anatomical sti-ucture. 
Still other changes will be obvious to a person skilled in the art. 

Advantages Of The Pip^nu Invenlinn 

Nuincrous advantages arc obtained through the use of the present 
invention. 

For one thing, a surgical targeting system is provided to facilitate 
locating a particular anatomical stiucturc during a medical procedure. 

For another thing, a vidco-b.-iscd surgical targeting system is provided 
to facilitate locating a particular anatomical structure during a medical 
procedure. 

And a video-based surgical targeting system is provided which 
pcnnits a series of patient-specific 2-D images (obtained by scanning 
patient anatomy using one or more scanning devices of the type described 
above) to be assembled into a 3-D computer model of the patient's scanned 
structure. 

Also, a video-based surgical targeting system is provided which allows 
a physician to view the aforementioned patient-specific 2-D images on a 
display in any desired access sequence. 

Furthennore, a video-based surgical targeting system is provided 
which allows a physician to assemble a series of patient-specific 2-D 
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images inlo a palicnt-specific database, and ihcn to generate virtual images 
from the aforementioned patient-specific database, as seen from any desired 
virtual camera position, for viewing on a display. 

And a vidco-based surgical targeting syslcm is provided which allows 
a physician to generate virtual images from the aforementioned 3-D 
computer model, as seen from any desired virtual camera position, for 
viewing on a display. 

In addition, a video-based surgical targeting system is provided which 
permits a physician to place virtual planning markers about any sites of 
interest while viewing one or more of the aforementioned patient-specific 
2-D images, with those virtual planning markers then being incoiporated 
into the 3-D computer model, whereby those virtual planning markers can 
be displayed in their appropriate 3-D positions when generating virtual 
images of the 3-D computer model. 

And a video-based surgical targeting system is provided which pennits 
a physician to place virtual planning markers about any sites of interest 
while viewing virtual images of the 3-D computer model, with those virtual 
planning markers then being incoiporated into the 3-D computer model, 
whereby those virtual planning markers can be displayed in their 
appropriate positions when subsequently generating virtual images of the 
3-D computer model. 
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And a video-based surgical largeling syslein is provided which permits 
a physician to place virtual planning markers about any sites of interest 
while viewing virtual images of the 3-D compuler model, with those virtual 
plaiming markers then being incon)oratcd into the 3-D computer model and 
into llie database of 2-D images, whereby those virtual planning markers 
can be displayed in their appropriate positions when subsequently 
generating virtual images of the 3-D computer model or when subsequently 
displaying 2-D images from the patient-specific database. 

For another thing, a video-based surgical targeting system is provided 
which pennits a real image obtained by a real-time imaging device (e.g. a 
video camera) to be displayed to a physician, and which pennits a virtual 
image generated from the 3-D computer model to be disj)layed to a 
physician, according to the directive of the physician. 

Aiid a video-based surgical targeting system is provided which pennits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be appropriately merged with a corresponding virtual image generated from 
the 3-D computer model. 

Also, a video-based surgical targeting system is provided which 
permits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be appropriately merged with a conesponding virtual image 
generated from the 3-D computer model, whereby the two images will be in 
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registralion with one another. 

Furthennore, a video-based surgical largeling system is provided 
which permits a real image obtained by a real-time imaging device (e.g. a 
video camera) to be merged with a con esponding virtual image generated 
from llie 3-D computer model, wiicreby the two images will be in 
registration with one another, and whereby the physician can choose to 
display either one of the two images exclusive of the other, or a composite 
of both images simultaneously. 

And a video-based surgical targeting system is provided which pennits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged with a corresponding viilual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one another, and whereby the physician can modify the 
virtual image generated from the 3-D computer model as needed, by 
clipping or fading, so as to expose the virtual planning markers to view, 
with the virtual planning markers being superimposed on the real image 
generated by the real-time imaging device. 

And a video-based surgical targeting system is provided which permits 
a real image generated by a real-time imaging device (e.g. a video camera) 
to be merged with a coiiesponding virtual image generated Hon) the 3-D 
compulcr model, whereby the two images will be simultaneously displayed 
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in registration willi one another, and wJiercby Ihe physician can modify ihc 
virtual image generated from the 3-D computer model so as to expose only 
tlie virtual planning markers to view, with the virtual planning markers 
being superimposed on the real image generated by the real-lime imaging 
device. 

Moreover, a video-based surgical targeting system is provided which 
pennits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be merged with a coiresponding virtual image generated from 
Uie 3-D computer model, whereby the two images will be simultaneously 
displayed in registration with one another, and whereby this registration 
will be automatically maintained even as the real-time imaging device is 
moved about relative to the anatomical site, with the virtual image being 
automatically generated so as to follow the real image. 

And a video-based surgical targeting system is provided which pennits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one another, and whereby this registration will be 
automatically maintained through the use of a computerized position and 
orientation tracker connected to the imaging device even as the real-time 
imaging device is moved about relative to the anatomical site, with the 
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virtual image being automatically generated so as to follow the real image. 

Also, a video-based surgical largcling system is provided which 
permits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be merged with a conesponding virtual image generated from 
Uie 3-D computer model, whereby the two images will be simultaneously 
displayed in registration with one another, and whereby this registration 
will be automatically maintained through the use of a computer search 
algorithm based on the real image and the virtual image even as the 
real-time imaging device is moved about relnlivc to the nnalonncal site, 
with the virtual image being automatically generated so as to follow the real 
image. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one another, and whereby a surgical instiiimenl can be 
tracked along with the real-time imaging device in order that the surgical 
instrument can be guided about the anatomical site using the virtual image 
generated from the 3-D computer model. 

In addition, a video-based surgical targeting system is provided which 
permits a virtual image to be generated from the 3-D computer model, and 
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further wherein Ihis virtual image can coiiespond lo an "over the shoulder 
view of Uie working lip of a surgical inslruinent located at llie anatomical 
site. 

And an improved method for locating anatomical structures during a 
medical procedure is provided. 
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In the preceding description^ data acquisition 
device 5 was used to generate patient-specific 2-D 
images, and these images were stored in 
patient-specific database 10; the patient-specific 2-D 
images stored in patient-specific database 10 were used 
to build patient-specific 3-D computer model 15 
viewable with image generator 30; and the output of 
image generator 30 and video source 45 were connected 
through video mixing device 35, whereby the surgeon can 
selectively display (i) a virtual image created by 
image generator 30 from patient-specific 3-D computer 
model, 15, or (ii) a real image obtained from video 
source 45, or (iii) a composite of the virtual image 
and the real image, where the virtual image is 
superimposed against the real image. 

In addition, means 20 are provided for the 
placement of virtual planning markers 25 into the 
patient-specific data; this may be done by placing 
such virtual planning markers into one or more of the 
patient-specific 2-D images contained in 
patient-specific database 10, or by placing such 
virtual planning markers into patient-specific 3-D 
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computer model 15, or by some combination of the two. 
In accordance with one aspect of the invention, virtual 
planning markers 25 placed in the patient-specific 2-D 
images stored in patient-specific database 10 will be 
automatically incorporated into patient-specific 3-D 
computer model 15, and virtual planning markers 25 
placed into patient-specific 3-D computer model 15 will 
be automatically incorporated into the patient-specific 
2-D images stored in patient-specific database 10. 

In this respect it should be appreciated that 
video source 45 may comprise a video camera, an 
endoscope, an ultrasound device, a fluoroscope, or any 
other type of appropriate image capturing means. 

It should also be appreciated that it is possible 
to create virtual planning markers 25 having a form 
other than the sphere-like markers shown in Fig. 9. 

By way of example, and looking now at Fig. 11, 
there is shown the aorta 105 of a patient as generated 
by a video source 45, e.g., a fluoroscope. Also shown 
is a series of sphere-like markers 25A placed into the 
system (e.g., by a human operator using a mouse) and a 
series of line segments 25B extending between selected 
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ones of the sphere-like markers 25A. These sphere-like 
markers 25A and line segments 25B may be used to plan a 
surgical procedure, to determine anatomical lengths or 
angles, etc. 

Also shown is a straight tube 25C which may also 
be used for planning and measurement purposes, etc., a 
curved tube 25D which may be used for planning and 
measurement purposes, and a box 25E which may be used 
for planning and measurement purposes, e.g., for volume 
calculations. 

In addition, other geometric elements such as 
curved lines, intersecting lines, etc. may also be 
provided for planning and measurement purposes. 

Significantly, it is also possible to insert into 
the system virtual planning markers representing 
virtual grafts, virtual implants, etc. By way of 
example, in Fig. 12 there is shown a virtual graft 25F 
which represents an arterial stent which may be 
deployed in the aorta, e.g., to treat an aortic 
aneurysm. 

Fig. 12 is a view showing a virtual graft 25F 
positioned on aorta 105. 
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